CURLOPT_MAX_RECV_SPEED_LARGE < CURLOPT_BUFFERSIZE breaks SFTP download

[alervd]

I did this

1. Create and setup an easy handle for SFTP download.
2. Call this: curl_easy_setopt(easy, CURLOPT_MAX_RECV_SPEED_LARGE, (curl_off_t)1000).
3. Create multi handle and add easy handle to it.
4. Start download. Result: it downloads first 1000 bytes and stalls. No further downloading occurs. Subsequent calls to turn the speed limit off has no effect also (curl_easy_setopt(easy, CURLOPT_MAX_RECV_SPEED_LARGE, (curl_off_t)0)).

I expected the following

I expect it to download with the speed limit I set with no limitations and to be able to change this limit at any time.

curl/libcurl version

libcurl 8.7.1.

operating system

Windows 10 22H2.

[bagder]

I cannot reproduce using the curl tool on Linux. Tested 8.7.1 and git master:

curl --limit-rate 1000 https://curl.se/

[alervd]

@bagder Well, it seems it's protocol specific. I can't reproduce it for HTTP protocol too. For me, it's reproducible for SFTP protocol.

[alexdconf]

Are you using your curl command in Powershell, CMD prompt, WSL -- on your Win10 install -- can you explicitly tell us which?
If you are not using WSL, can you post the output of the command you use to check the version of curl you are using?

On Windows systems, sometimes commands such as curl are aliased and actually implemented, not by the open source project, but some MS implementation -- just want to check that.

[alervd]

@alexdconf I'm using libcurl in my own code and have this issue. However, I've just tried to test it with curl from official downloads and it has the same issue for SFTP link. I launched it from CMD, however I'm pretty sure it does not matter at all (cmd, powershell, etc.).

[alexdconf]

@alervd Apologies, I missed that you were working with the library directly and not in a CLI environment — since you cited Windows, I know that curl is aliased in those systems sometimes so I wanted to be sure you were leveraging the actual curl.

I think I managed to reproduce the problem you have.

I’m on a M1 Mac so I used a Docker container with a fedora:latest base image to reproduce this. I also spun up a sftp container to test this containing the file (just some file I had) BayesMask1.png in the /home/ directory.

The test script in the Fedora environment:

#!/bin/bash
curl -V
curl --limit-rate 1000 --max-time 60 -O -k -u "sftp:blank" sftp://10.0.0.5/~/BayesMask1.png
echo "Done."

The --max-time 60 is there because without it, the curl command hung for over fifteen minutes and I wanted to get logs. Without the --limit-rate 1000 option, the file downloads properly, fairly immediately.

The output from the script in the Fedora environment:

2024-05-18 11:46:03 curl 8.6.0 (aarch64-redhat-linux-gnu) libcurl/8.6.0 OpenSSL/3.2.1 zlib/1.3.0.zlib-ng brotli/1.1.0 libidn2/2.3.7 libpsl/0.21.5 libssh/0.10.6/openssl/zlib nghttp2/1.59.0 OpenLDAP/2.6.7
2024-05-18 11:46:03 Release-Date: 2024-01-31
2024-05-18 11:46:03 Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
2024-05-18 11:46:03 Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets
2024-05-18 11:46:03   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2024-05-18 11:46:03                                  Dload  Upload   Total   Spent    Left  Speed
  3 25806    3  1000    0     0     16      0  0:26:52  0:00:59  0:25:53     0
  3 25806    3  1000    0     0     16      0  0:26:52  0:00:59  0:25:53    16
2024-05-18 11:47:03 curl: (28) Operation timed out after 59999 milliseconds with 1000 out of 25806 bytes received
2024-05-18 11:47:03 Done.

[bagder]

it's reproducible for SFTP protocol

What SSH library and version are you using? (curl -V output is always a good idea...)

[alervd]

@bagder

curl 8.7.1 (x86_64-w64-mingw32) libcurl/8.7.1 LibreSSL/3.9.1 zlib/1.3.1 brotli/1.1.0 zstd/1.5.6 WinIDN libpsl/0.21.5 libssh2/1.11.0 nghttp2/1.61.0 ngtcp2/1.4.0 nghttp3/1.2.0
Release-Date: 2024-03-27
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL SSPI threadsafe UnixSockets zstd

[alervd]

Any ETA on this issue fix? I've found that libcurl is too slow downloading from SFTP in case CURLOPT_BUFFERSIZE is set to default 16K. Now I have to increase it to 256K and found that my minimum speed limit is increased to this 256K also... :)

[alervd]

For now, it seems there is a workaround (return CURL_WRITEFUNC_PAUSE in write func; didn't try it btw). But I would prefer to wait for a fix in case it appears in some reasonable time.

[bagder]

Any ETA on this issue fix?

I have not seen anyone working on this. There is no estimate.